Characterizing concurrency mechanisms for NVIDIA GPUs under deep learning workloads

We investigate the performance of concurrency mechanisms available on NVIDIA’s new Ampere GPU microarchitecture under deep learning training and inference workloads. In contrast to previous studies that treat as a black box, we examine scheduling at microarchitectural level. find lack fine-grained preemption mechanisms, robust task prioritization options, contention-aware thread block placement policies limits effectiveness mechanisms. summary, sequential nature workloads their fluctuating resource requirements kernel runtimes make executing such while maintaining consistently high utilization low, predictable turnaround times difficult current NVIDIA hardware.